This invention relates generally to software, and more particularly to a method and system for automating the collection of diagnostic traces once a failure has been detected in a software automation test framework, prior to a test environment being torn down.
Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, the setting up of test preconditions, and other test control and test reporting functions. Commonly, test automation involves automating a manual process already in place that uses a formalized testing process. Test automation tools assist programmers to quickly create applications that have dramatically improved programmer productivity, as well as decreasing the pressure on testers, who are often perceived as bottlenecks to the delivery of software products. Testers are consistently asked to test more and more programming code in less and less time. Test automation is one way to keep up with the increasing amounts of programming code that requires testing, as manual testing is time consuming. Traditionally, as different versions of software are released, the new features have to be tested manually. However, test automation tools reduce the test time, as well as the cost associated with new software releases.
A framework is an integrated system that sets the rules of automation of a specific product. A framework integrates the function libraries, test data sources, object details and various reusable modules. The components that make up the framework act as small building blocks that need to be assembled in a regular fashion to represent a business process. In the implementation of test automation, a framework may contain a common set of tools, which may consist of a combination of automation scripts to configure resources needed for the test application, and an environment in which to execute the scripts. Thus, frameworks provide the basis of test automation, and hence simplify the automation effort.
A growing trend in software development is to use testing frameworks such as the xUnit frameworks (for example, JUnit and NUnit) which allow the framework code to conduct unit tests to determine whether various sections of new code are acting as expected under various circumstances. Test cases describe tests that need to be run on the program to verify that the program runs as expected.
JUnit is a unit testing framework for the Java programming language. In computer programming, unit testing is a method of testing that verifies the individual units of source code are working properly. Unit testing is typically done by software developers to ensure that the code they have written meets software requirements, and behaves as the developer intended. A unit is the smallest testable part of an application. The goal of unit testing is to isolate each part of the program, and show that the individual parts are correct. Unit testing allows the programmer to refactor code at a later date, and make sure the module still works correctly (i.e., regression testing). Unit testing helps to eliminate uncertainty in the units themselves and may be used in a bottom-up testing style approach. By testing the parts of a program first and then testing the sum of its parts, integration testing becomes much easier.
The procedure for unit testing is to write test cases for all functions and methods so that whenever a change causes a fault, it may be quickly identified and fixed. A unit test provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits. Unit tests find problems early in the development cycle. In procedural programming a unit may be an individual program, function, procedure, etc., while in object-oriented programming, the smallest unit is a method, which may belong to a base/super class, abstract class or derived/child class. Ideally, each test case is independent from the others; double objects like stubs, mock or fake objects as well as test harnesses may be used to assist testing a module in isolation.
In software testing, a test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. A test harness has two main parts: a test execution engine and a test script repository. Test harnesses allow for the automation of tests. Test harnesses may call functions with supplied parameters, and print out and compare the results to the desired value. A test harness is a hook to the developed code, which may be tested using an automation framework. A test harness should allow specific tests to run (this helps in optimizing), orchestrate a runtime environment, and provide a capability to analyze results. The typical objectives of a test harness are to automate the testing process, execute test suites of test cases, and generate associated test reports. A test harness typically provides the following benefits including, increased productivity due to automation of the testing process, increased probability that regression testing will occur, and increased quality of software components and application.